Executing A Selected Sequence Of Instructions Depending On Packet Type In An Exact-Match Flow Switch

ABSTRACT

An integrated circuit includes a processor and an exact-match flow table structure. A first packet is received onto the integrated circuit. The packet is determined to be of a first type. As a result of this determination, execution by the processor of a first sequence of instructions is initiated. This execution causes bits of the first packet to be concatenated and modified in a first way, thereby generating a first Flow Id. The first Flow Id is an exact-match for the Flow Id of a first stored flow entry. A second packet is received. It is of a first type. As a result, a second sequence of instructions is executed. This causes bits of the second packet to be concatenated and modified in a second way, thereby generating a second Flow Id. The second Flow Id is an exact-match for the Flow Id of a second stored flow entry.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims priority under 35U.S.C. § 120 from, nonprovisional U.S. patent application Ser. No.14/726,438 entitled “Executing A Selected Sequence Of InstructionsDepending On Packet Type In An Exact-Match Flow Switch,” filed on May29, 2015, now U.S. Pat. No. 10,033,638. The disclosure of the foregoingdocument is incorporated herein by reference.

TECHNICAL FIELD

The described embodiments relate generally to structures and methods forswitching packets using a Flow ID and an exact-match flow table.

SUMMARY

In a first novel aspect, an integrated circuit includes an exact-matchflow table structure. A Static Random Access Memory (SRAM) within theexact-match flow table structure stores a novel exact-match flow table.The exact-match flow table is a flow table for storing flow entries,where each flow entry includes a Flow Identifier value (Flow Id) and anassociated action value. Each Flow Id is a multi-bit digital value, eachbit of which is efficiently stored in one and only one cell of the SRAM.The exact-match flow table structure does not, and cannot, store awildcard bit indictor or mask value. The exact-match flow tablestructure stores no indicator that any particular part of a packetshould be matched against any particular part of a Flow Id. The Flow Idis not a flow entry of an OpenFlow flow table. The Flow Id uniquelyidentifies one flow of packets.

In one example, an incoming packet is received onto the integratedcircuit and the integrated circuit generates a Flow Id from the incomingpacket. The generated Flow Id is supplied to the exact-match flow tablestructure, and the exact-match flow table structure determines whetherthe generated Flow Id is a bit-by-bit exact-match of any Flow Id of anyflow entry stored in the SRAM. If the determination is that thegenerated Flow Id is a bit-by-bit exact-match of a Flow Id of a storedflow entry, then the exact-match flow table structure outputs the actionvalue that was stored along with the matching Flow Id. Circuitry on theintegrated circuit then performs an action indicated by the actionvalue. In one example, this action is to output the packet from theintegrated circuit via an egress port indicated by the action value. Theexact-match flow table structure stores at most one Flow Id that canmatch a Flow Id generated due to the receiving of a legal packet ontothe integrated circuit.

If, on the other hand, the determination is that the generated Flow Idis not a bit-by-bit exact-match of any Flow Id of any stored flow entry,then a miss indication communication is output from the integratedcircuit. In one example, in a miss situation, the exact-match flow tablestructure outputs a default action value. The default action valuecauses the packet to be sent out of an egress port indicated by thedefault action value so that the packet is then communicated from theintegrated circuit to a Network Flow Processor (NFP) integrated circuit.The packet is communicated from the integrated circuit to the NFP inencapsulated form. The NFP receives the packet, and determines how itwants the integrated circuit to handle other packets of this flow. Inone example, the NFP sends a command communication back to theintegrated circuit, where the command communication includes a new FlowId. The command communication is a command to the integrated circuit toload the new Flow Id into the exact-match flow table. Thereafter, asubsequent packet of the same flow is received onto the integratedcircuit. The integrated circuit goes through the Flow Id generatingprocess and generates a Flow Id for the subsequent packet. The Flow Idis presented to the exact-match flow table, but now there is abit-by-bit matching Flow Id stored in the exact-match flow table. Theaction indicated by the action value of the matching flow entry is thencarried out. In one example, the action is to output the second packetfrom the integrated circuit on a particular output port of theintegrated circuit. In this way, subsequent packets of the flow will beoutput by the integrated circuit in accordance with the action value ofthe newly added flow entry. An action value can be an action toencapsulate an incoming packet or packet information, in addition tosending the encapsulated packet out of a particular egress port.

In a second novel aspect, a Flow Id is generated using a novelmultiplexer circuit. In one example, the multiplexer circuit includes aplurality of byte-wide multiplexer circuits. Each byte-wide multiplexercircuit has a plurality of sets of input leads, a set of select inputleads, and a set of output leads. A first of the byte-wide multiplexercircuits outputs a byte that is the first byte of the Flow Id, a secondof the byte-wide multiplexer circuits outputs a byte that is the secondbyte of the Flow Id, a third of the byte-wide multiplexer circuitsoutputs a byte that is the third byte of the Flow Id, and so forth. Eachrespective set of input leads of each byte-wide multiplexer circuit iscoupled to receive a corresponding respective byte of the incomingpacket, or other data, or other modified or compressed packet data. Bycontrolling the select values supplied onto the set of select inputleads of a byte-wide multiplexer circuit, the byte-wide multiplexercircuit is made to select an appropriate byte (of incoming packet data,or other data, or modified or compressed packet data). In one novelmethod, a packet is received onto an input port of the integratedcircuit. A characterizer and classifier analyzes the packet andclassifies the packet as being in one of a plurality of classes. If theincoming packet is determined to be in a first class then the byte-widemultiplexer circuits are controlled in a first way such that a first setof selected bytes is output by the multiplexer circuit in a particularbyte order. The bytes output by the multiplexer circuit are a Flow Id.If, however, the incoming packet is determined to be in a second classthen the byte-wide multiplexer circuits are controlled in a second waysuch that a second set of selected bytes is output by the multiplexercircuit in a particular byte order. The bytes output by the multiplexercircuit are a Flow Id. The generated Flow Id (which is generated in aselected way of a plurality of ways, as determined by the type of thepacket) is supplied from the multiplexer circuit and to the exact-matchflow table structure.

In a third novel aspect, a Flow Id is generated using a programmablereduce table circuit. An incoming packet is received onto the integratedcircuit that includes the exact-match flow table structure. A Flow Id isgenerated from the packet by using a selected subset of bits of theincoming packet as address bits to address a reduce table SRAM, suchthat the reduce table SRAM outputs a multi-bit data value. A barrelshifter and mask circuit and adder circuit are used, in one example, togenerate the bits that are supplied as the address to the reduce tableSRAM. The multi-bit data value that is output from the reduce table SRAMis supplied to a programmable lookup circuit, that in turn performs aselected one of a plurality of lookup operations, and outputs a lookupresult value. The lookup result is of a reduced number of bits. Amultiplexer circuit (for example, a multiplexer circuit involving aplurality of byte-wide multiplexer circuits as set forth above) receivesthe lookup result value and outputs at least a part of the lookup resultvalue so that at least a part of the result value is a part of a Flow Idoutput by the multiplexer circuit. In one example, output leads of thelookup circuit supply one or more bytes onto a particular one or more ofthe sets of input leads of the multiplexer circuit. By appropriatecontrol of the select input leads of the multiplexer circuit, themultiplexer circuit is made to output a part or all of the lookup resultvalue as part of the Flow Id. In one example, the programmable lookupcircuit is controlled to perform a first type of lookup operation if theincoming packet is determined to be of a first type, whereas theprogrammable lookup circuit is controlled to perform a second type oflookup operation if the incoming packet is determined to be of a secondtype. In a similar way, an incoming packet can be determined to be ofone of three or more types, and the programmable lookup circuit can becontrolled to carry out a selected one of three or more lookupoperations. In another example, an additional novel template SRAM isprovided. The template SRAM receives a template value from theclassifier, and outputs bits that control how the reduce table SRAM isaddressed, which lookup operation will be performed by the programmablelookup circuit, and how the select input of the multiplexer circuit willbe controlled.

In a fourth novel aspect, an integrated circuit that has an exact-matchflow table structure also has a crossbar switch, a script-based egresspacket modifier, en egress packet script memory, a first egress packetmodifier memory, and a second egress packet memory. The egress packetscript memory stores a plurality of scripts. The first egress packetmodifier memory stores non-flow specific header information. The secondegress packet modifier memory stores flow specific header information.In one novel method, a packet is received onto the integrated circuitand a Flow Id is generated from the packet as described above. The FlowId is supplied to an exact-match flow table structure as describedabove, and an exact-match is found. The flow entry for which the Flow Idis a bit-by-bit exact-match includes, in addition to the Flow Id, anegress action value, an egress flow number, and an egress port number.The egress action value and the egress flow number are then communicatedacross the crossbar switch along with a portion (not all) of the packet.The egress action value, the egress flow number and the portion of thepacket are received from the crossbar switch and onto the script-basedegress packet modifier. The script-based egress packet modifier uses theegress action value to retrieve an indicated script from the egresspacket script memory. A script interpreter of the egress packet modifierinterprets the script. Interpreting of the script causes the egresspacket modifier to retrieve non-flow specific header information fromthe first egress packet modifier memory. In addition, the egress packetmodifier uses the egress flow number to retrieve flow specific headerinformation from the second egress packet modifier memory. The egresspacket modifier adds the non-flow specific header information and theflow specific header information onto the front of the portion of thepacket, thereby recreating appropriate header fields, and therebygenerating a complete output packet. The output packet is then outputfrom the integrated circuit via the particular egress port indicated bythe egress port number. By storing the non-flow specific and flowspecific information in memories local to the egress packet modifier,and by not sending this header information across the crossbar switchwith the packet, the amount of switching and congestion occurring in thecrossbar switch is reduced, and power consumption in the crossbar switchis reduced. By storing the non-flow specific and the flow specificheader information in different memories, the total amount of memoryrequired to store the necessary header information is reduced ascompared to using a single memory in a conventional manner. One reasonfor including the crossbar switch is to facilitate scaling largerintegrated circuits from subcircuits (e.g. an NFX integrated circuit).

In a fifth novel aspect, an integrated circuit includes an exact-matchflow table structure and a processor. In one example, the processor is avery small and efficient run-to-completion picoengine processor thatdoes not include an instruction counter and that does not fetchinstructions unless it is externally prompted to do so. A first packetis received onto the integrated circuit and is analyzed and isdetermined to be of a first type. As a result of this determination,execution by the processor of a first sequence of instructions isinitiated. Execution of the first sequence causes bits of the firstpacket to be concatenated and modified in a first way thereby generatinga first Flow Id of a first form. The first Flow Id of the first form isa bit-by-bit exact-match of a Flow Id of a first flow entry stored inthe exact-match flow table maintained in the exact-match flow tablestructure. The first packet is output from the integrated circuit inaccordance with an action value of the matching first flow entry. Asecond packet is received onto the integrated circuit and is analyzedand is determined to be of a second type. As a result of thisdetermination, execution by the processor of a second sequence ofinstructions is initiated. Execution of the second sequence causes bitsof the second packet to be concatenated and modified in a second waythereby generating a second Flow Id of a second form. The second Flow Idof the second form is a bit-by-bit exact-match of a Flow Id of a secondflow entry stored in the exact-match flow table. The second packet isoutput from the integrated circuit in accordance with an action value ofthe matching second flow entry. In one example, execution by thepicoengine processor of the first sequence causes a programmable lookupcircuit and a multiplexer circuit to be controlled in a first way sothat a Flow Id of a first form is generated, whereas execution by thepicoengine processor of the second sequence causes a programmable lookupcircuit and a multiplexer circuit to be controlled in a second way sothat a Flow Id of a second form is generated. The exact-match flow tablestructure can store and match to Flow Ids of both forms.

In a sixth novel aspect, an integrated circuit comprises an input port,a first characterize/Classify/Table Lookup and Multiplexer Circuit(CCTC), a second CCTC, and an exact-match flow table structure. Thesecond CCTC is substantially structurally identical to the first CCTC.The first CCCT and the second CCTC are coupled together serially so thatpacket information flows serially from the input port, through the firstCCTC, through the second CCTC, and to the exact-match flow tablestructure. In one example, an incoming packet is received onto theintegrated circuit via the input port. A first characterizer of thefirst CCTC receives the incoming packet and outputs a characterizationoutput value. A first classifier of the first CCTC receives thecharacterization output value and generates a data output value and acontrol information value. A first Table Lookup and Multiplexer Circuit(TLMC) of the first CCTC is coupled to receive the data output value andthe control information value from the first classifier. A programmablelookup circuit of the first TLMC is controllable to receive a selectablepart of the data output value and to output an associated lookup value.A multiplexer circuit of the first TLMC is controllable to output thelookup value as a part of a data output value. The second CCTC includesa second characterizer, a second classifier, and a second TLMC. Thesecond characterizer of the second CCTC receives at least a part of thedata output value that is output by the first TLMC. The secondclassifier is coupled to receive a characterization output value fromthe second characterizer. The second TLMC is coupled to receive the dataoutput value from the second classifier and to receive a controlinformation value from the second classifier. A programmable lookupcircuit of the second TLMC is controllable to receive a selectable partof the data output value received from the second classifier and tooutput an associated lookup value. A multiplexer circuit of the secondCCTC is controllable to output the lookup value as a part of a Flow Id.The Flow Id as output by the second CCTC is supplied to the exact-matchflow table structure, which in turn determines whether any Flow Id ofany flow entry stored in the exact-match flow table structure is abit-by-bit exact-match of the Flow Id as output by the second CCTC.

Further details and embodiments and methods and techniques are describedin the detailed description below. This summary does not purport todefine the invention. The invention is defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, where like numerals indicate like components,illustrate embodiments of the invention.

FIG. 1 is a diagram of a data center in accordance with one novelaspect.

FIG. 2 is a diagram of a rack of network devices within the data centerof FIG. 3.

FIG. 3 is a block diagram of an SDN switch in the rack of FIG. 2.

FIG. 4 is a block diagram of one of the NFX circuits in the SDN switchof FIG. 3.

FIG. 5 is a more detailed diagram of one of the corner portions of theNFX circuit of FIG. 4.

FIG. 6 is a more detailed diagram of the exact-match flow tablestructure shown in FIG. 5.

FIG. 7 is a more detailed diagram of Characterize/Classify/Table Lookupand Mux (CCT) circuit 115 of FIG. 5.

FIG. 8 is a more detailed diagram of characterizer 200.

FIG. 9 is a table illustrating the contents of the characterizationresult.

FIG. 10 is a diagram of the contents of template RAM 221.

FIG. 11 is a more detailed diagram of mux A 223.

FIG. 12 is a more detailed diagram of reduce table A 224.

FIG. 13 is a more detailed diagram of multiplexer circuit 226.

FIG. 14 is a more detailed diagram of a picoengine pool.

FIG. 15 is a diagram illustrating one sequence of instructions that canbe executed by a picoengine.

FIG. 16 is a more detailed diagram of egress modifier circuit 147.

FIG. 17 is a more detailed diagram of the data center 10 of FIG. 1.

FIG. 18 is a diagram of a communication from the gateway to the virtualmachine in the data center.

FIG. 19 is diagram illustrating the conversion from packet type B to apacket type C.

FIG. 20 is a diagram of a communication between a first virtual machineand a second virtual machine through a single SDN switch.

FIG. 21 is a diagram of a communication from a virtual machine through aSDN switch and a spine to a gateway.

FIG. 22A is a diagram illustrating a conversion from a packet type C toa packet type B.

FIG. 22B is a diagram illustrating a conversion from a packet type C toa packet type B.

FIG. 23A is a flowchart illustrating the steps of VXLAN encapsulationand flow routing in a SDN switch (leaf).

FIG. 23B is a flowchart illustrating the steps of VXLAN encapsulationand flow routing in a SDN switch (leaf).

FIG. 24 is a diagram of an input to a reduce table A.

FIG. 25 is a diagram of an output of a barrel shifter included in reducetable A.

FIG. 26 is diagram of an output from reduce table A.

FIG. 27 is a diagram of an input to reduce table B.

FIG. 28 is a diagram of an output of a barrel shifter included in reducetable B.

FIG. 29 is a diagram of an output from reduce table B.

FIG. 30 is a diagram of a flow ID.

FIG. 31 is a diagram of the flow entry that is stored in an entry fieldin the flow table of the corner portion of FIG. 5.

FIG. 32 is a diagram illustrating a bit-by-bit match using anexact-match flow table.

FIG. 33 is a diagram illustrating a bit-by-bit match using anexact-match flow table on a single integrated circuit.

FIG. 34 is a diagram illustrating the contents of a crossbarcommunication.

FIG. 35 is a diagram illustrating the contents of one set of argumentsread from the argument A SRAM in the egress modifier.

FIG. 36 is a diagram illustrating the contents of one set of argumentsread from the argument B SRAM in the egress modifier.

FIG. 37 is a diagram of a MAC frame, such as a MAC frame received ontoan NFX port, when the port is operating in the command mode.

FIG. 38 is a diagram that illustrates the format of one of the commandheaders in the command of the MAC frame of FIG. 37.

FIG. 39 is a table that sets forth the various opcodes possible in acommand header.

DETAILED DESCRIPTION

Reference will now be made in detail to background examples and someembodiments of the invention, examples of which are illustrated in theaccompanying drawings.

FIG. 1 is a diagram of a system commonly referred to as a “data center”10. Data center 10 in this example includes multiple racks 11-15 ofnetworking devices, multiple spines 16-19, and a gateway device 20.External internet traffic from and to the internet 28 enters and exitsthe data center via the gateway device 20. For purposes of switchingpackets within the data center, the gateway may add a tag onto eachpacket as the packet enters the data center. The tags are usedinternally within the data center in accordance with an SDN(Software-Defined Networking) protocol. The gateway strips the tags offthe packets when the packets leave the data center.

FIG. 2 is a more detailed diagram of one of the racks 11 of FIG. 1. Therack 11 is a physical support structure that holds a stack ofrectangularly-shaped networking devices. The upper networking device isa SDN switch 21 referred to as a “top-of-rack” switch or “leaf”.Although the SDN switch 21 is referred to as a “top-of-rack” switch, theSDN switch that performs this function and that is interconnected inthis way need not actually be located in the uppermost slot. The SDNswitch 21 involves a set of printed circuit boards and an associatedinterconnect backplane that are disposed in a rectangular box-likeenclosure. The SDN switch 21 slides into the uppermost slot in the rackand is held in place and is physically supported by the rack. Multiplehost server devices 22-26 are also held by the rack in slot positionsbelow the top-of-rack switch. The top-of-rack switch 21 is coupled toeach of the host server devices 22-26 by networking cables. A networkingcable has a first plug that on one end of the cable plugs into a networkport socket located on the back of the SDN switch 21 and also has asecond plug that on the other end of the cable plugs into a network portsocket located on the back of one of the server devices. The cable plugsand sockets are not illustrated in FIG. 2. As shown in FIG. 1, thetop-of-rack SDN switches of the data center are typically not coupleddirectly to one another by network cables, but rather they maycommunicate with each other via appropriate ones of the spines asillustrated. The vertical lines 27 illustrated in FIG. 1 represent manynetworking cables that link the top-of-rack SDN switch 21 to the variousserver devices 22-26 of the same rack. Each of the racks of componentsis of the same structure.

FIG. 3 is a more detailed diagram of SDN switch 21. SDN switch 21includes a plurality of QSFP 40 Gbps modules 29-48, four Network FlowSwitch (NFX) circuits 49-52, a Network Flow Processor (NFP) circuit 53,and a control processor circuit 54. These electronic components aredisposed on a set of printed circuit boards. The printed circuit boardsare interconnected by a backplane. The printed circuit boards and thebackplane are disposed in a box-like enclosure.

Each QSFP module has a socket for receiving a plug. The QSFP modules aredisposed on the printed circuit boards so that the sockets areaccessible from outside the enclosure. Each QSFP module providestransceivers for four outgoing optical fibers, and for four incomingoptical fibers. Each optical fiber provides 10 Gbps communication, sothe overall QSFP module provides 40 Gbps communication in bothdirections. The eight optical fibers are parts of a single optical cablethat has plugs on either end to plug into QSFP modules sockets.

Control processor circuit 54 is a processor system that includes, amongother parts not illustrated, a control processor integrated circuit 55and an amount of memory 56. The control processor integrated circuit 55in one example is a CPU processor integrated circuit such as anx86-architecture processor that runs a standard operating system, andthat has PCIe bus interfaces 60 and 61 for communicating with otherdevices that have PCIe communication capabilities. A SDN protocol stack57 is a part of, or is executing on, the operating system. In addition,the control processor circuit 54 stores an SDN flow table 58. The SDNflow table 58 is stored in a combination of the memory 56 and theprocessor 55. In the present example, the SDN protocol stack 57 is anOpenFlow protocol stack that is compliant with the OpenFlow SwitchSpecification, version 1.1.0, Implemented (Feb. 28, 2011). The OpenFlowprotocol stack 57 causes OpenFlow flow entries to be added into, and tobe deleted from, the SDN flow table 58. The OpenFlow protocol stack 57can receive and handle OpenFlow compliant messages. The OpenFlowprotocol stack also generates and outputs OpenFlow messages inaccordance with the OpenFlow standard.

Network Flow Processor (NFP) circuit 53 is a processor system thatincludes, among other parts not illustrated, a NFP integrated circuit 62and an amount of external memory 63. Of importance, the NFP circuit 53does not execute any standard operating system, and the NFP circuit 53does not execute any SDN protocol stack. In the specific example setforth here, the NFP integrated circuit 62 is an instance of theIsland-Based Network Flow Processor integrated circuit set forth in U.S.patent application Ser. No. 13/399,888, entitled “Island-Based NetworkFlow Processor Integrated Circuit”, filed Feb. 17, 2012, by Gavin J.Stark et al. (the subject matter of which is incorporated herein byreference). Although the NFP circuit 53 does not execute an SDN protocolstack, and does not execute any standard operating system of the typethat would typically support an OpenFlow protocol stack, the NFP circuit53 does maintain a copy of the SDN table 58. The copy of the SDN table58 maintained by the NFP circuit 53 is identified by reference numeral64. In addition, that NFP circuit 53 maintains a copy of a set ofsixteen non-SDN flow tables 65-80. The NFP integrated circuit 62 hasPCIe bus interfaces 81 and 82. Lines 83 and 84 represent a PCIe serialbus 85 that couples the control processor circuit 54 and the NFP circuit53.

In addition to PCIe bus interfaces, the NFP integrated circuit 62 alsohas a set of network ports 86-89. Each of these network ports is a 40Gbps bidirectional ethernet network port. The ingress portion of theport involves SerDes circuits and an instance of the ingress MAC islandas set forth in U.S. patent application Ser. No. 14/321,732, entitled“Merging PCP Flows As They Are Assigned To A Single Virtual Channel”,filed Jul. 1, 2014, by Joseph M. Lamb (the subject matter of which isincorporated herein by reference). The egress portion of the portinvolves an instance of the egress MAC island and the associated SerDescircuits. A flow can contain a subflow. A flow and a subflow are bothconsidered a flow.

Each of the four Network Flow Switch (NFX) circuits is identical to theothers. In the example set forth here, an NFX circuit is an integratedcircuit that has twelve network ports denoted A1, A2, A3, B1, B2, B3 C1,C2, C3, D1, D2 and D3. In the diagram of FIG. 3, two of the networkports, namely ports A3 and B3, are not used or connected to, so theseports are not illustrated. Each of the network ports is a 40 Gbpsdirectional ethernet MAC port of the same structure and operation asports 86-89 of the NFP integrated circuit 53, except that the ingressportions of the network ports of the NFX circuits have an additionalcapability to handle special command packets as set forth in furtherdetail below. Of importance, none of the NFX circuits 49-52 stores ormaintains any SDN flow table, but rather the flow tables maintained byand stored on the NFX circuit are novel and special non-SDN flow tables.The upper-left four flow tables 65-68 on the NFP circuit 53 are copiesof the four non-SDN flow tables 90-93 stored in NFX circuit 49. Theupper-right four flow tables 69-72 on the NFP circuit 53 are copies ofthe four non-SDN flow tables 94-97 stored in NFX circuit 50. Thelower-left four flow tables 73-76 on the NFP circuit 53 are copies ofthe four non-SDN flow tables 98-101 stored in NFX circuit 51. Thelower-right four flow tables 77-80 on the NFP circuit 53 are copies ofthe four non-SDN flow tables 102-105 stored in NFX circuit 52. None ofthe NFX circuits executes any operating system, and none of the NFXcircuits has any instruction-fetching processor that fetchesinstructions, that has an instruction counter, and that analyzes anypacket. Compared to the relatively large and powerful and complex masterNFP integrated circuit 62 that may bring instruction-fetching processorsto bear on analyzing packets, the NFX circuits 49-52 are small andrelatively inexpensive slave switching circuits that perform directedswitching functions on behalf of the NFP circuit.

FIG. 4 is a more detailed diagram of NFX integrated circuit 49. The NFXintegrated circuit 49 is not included on a larger Network Flow Processor(NFP) integrated circuit and is much less expensive to manufacture thanthe cost to manufacture a NFP integrated circuit. As set forth above,all of the NFX circuits are identical integrated circuits in thisexample. NFX integrated circuit 49 includes four corner portions 106-109(denoted Corner A, Corner B, Corner C, and Corner D), a queue SRAM block110, a scheduler 111, and a central crossbar switch circuit 112. Eachcorner portion stores its own non-SDN flow table as illustrated. Eachcorner portion has three 40 Gbps ethernet MAC ingress ports and three 40Gbps ethernet MAC egress ports. For Corner A (106), the three ingressports are denoted A1I, A2I and A3I, and the four egress ports aredenoted A1O, A2O and A3O. Packet traffic coming into the NFX circuit isreceived via one of the ingress ports of a corner, and is communicatedto the crossbar switch circuit 112, and then if appropriate is switchedto exit the NFX circuit via an egress port of one of the cornerportions. The crossbar switch circuit 112 directs the packet to theappropriate corner portion from which the packet will exit the NFXcircuit.

In the specific embodiment of FIG. 4, the NFX integrated circuit 49 doesnot include any openflow flow tables and the NFX integrated circuit 49does not include any flow tables that include wildcards. The NFXintegrated circuit 49 never decides that a new flow entry is to be addedto the flow table, rather the decision/instruction to load a new flowentry into the flow table is always received from an external integratedcircuit and the exact flow entry to load into the flow table isgenerated by an external integrated circuit. The NFX integrated circuit49 only responds to external commands to load a new flow entry into theflow table. However, operation of the exact-match flow table in the NFXintegrated circuit 49 is consistent with a higher level openflow tablethat is not stored on the NFX integrated circuit 49 (e.g. a NFPintegrated circuit).

For additional information on NFX circuit 49, see: U.S. patentapplication Serial Numbers: 1) Ser. No. 14/634,844, entitled “A MODULARAND PARTITIONED SDN SWITCH”, 2) Ser. No. 14/634,845, entitled “METHOD OFHANDLING SDN PROTOCOL MESSAGES IN A MODULAR AND PARTITIONED SDN SWITCH”,3) Ser. No. 14/634,847, entitled “FORWARDING MESSAGES WITHIN A SWITCHFABRIC OF AN SDN SWITCH”, 4) Ser. No. 14/634,848, entitled “METHOD OFDETECTING LARGE FLOWS WITHIN A SWITCH FABRIC OF AN SDN SWITCH”, 5) Ser.No. 14/634,849, entitled “METHOD OF GENERATING SUBFLOW ENTRIES IN AN SDNSWITCH”, and 6) Ser. No. 14/634,851, entitled “SDN PROTOCOL MESSAGEHANDLING WITHIN A MODULAR AND PARTITIONED SDN SWITCH”, all of which werefiled Mar. 1, 2015, by Gavin J. Stark, (the subject matter of which isincorporated herein by reference).

FIG. 5 is a more detailed diagram of Corner A (106) of the NFX circuit49 of FIG. 4. The circuitry of each of the four corner portions of theNFX circuit 49 is identical to the circuitry of each other cornerportion. An incoming packet 113 is received onto one of the ingressports by ingress MAC circuit 114. For additional information on ingressMAC circuit 114, see: U.S. patent application Ser. No. 14/321,732,entitled “Merging PCP Flows As They Are Assigned To A Single VirtualChannel”, filed Jul. 1, 2014, by Joseph M. Lamb (the subject matter ofwhich is incorporated herein by reference).

In one specific example, each corner of the NFX includes twoCharacterize/Classify/Table Lookup and Mux circuits (CCTC) 115 and 116and an analysis type RAM 145. In one embodiment, analysis type RAM 145is a lookup memory. The output of ingress MAC 114 is provided toanalysis type RAM 145 and CCT circuit 115. In response analysis type RAM145 outputs an analysis type value. The analysis type value determinesthe type of analysis to be performed by the characterizer included inCCT circuit 115. In one novel aspect, the analysis type value is a 2-bitvalue that configures the CCT circuit to perform characterization of aspecific type of packet that is known to be received on a specific portnumber. In one example, the analysis type RAM 145 is programmed byanother integrated circuit. CCT circuits 115 and 116 are identicalcircuits. CCT circuit 115 is shown in greater detail in FIG. 7. CCTcircuit 115 includes a characterizer circuit 200, a classifier circuit201, and a table lookup and mux circuit 202. Characterizer 200 receivesis analysis type 203 from analysis type RAM 145, and input data value204 from ingress MAC 114, metadata 205 from a previous classifier (notapplicable to CCT circuit 115), and start of packet, end of packet,length of packet data 206 from ingress MAC 114. The characterizeroutputs the input data value 204, a characterization value 205, and thestart of packet, end of packet, and length of packet data 206 toclassifier 201.

FIG. 8 illustrates the characterizer 200 in greater detail. Thecharacterizer combines the analysis type value and the input data value(256 bits of an incoming packet) and generates a characterization of thepacket (characterization data). The analysis type is configured on alogical channel basis, that is, every logical channel can be assigned adifferent analysis type value to be used by the characterizer. Theanalysis type value indicates the first level of characterizationrequired for the packet. The characterizer 200 includes fourcharacterization stages: a channel configuration characterization stage330, an outer encapsulation characterization stage 331, an ethernetencapsulation characterization stage 332, and an inner packetcharacterization stage 333. The channel configuration characterizationstage 330 performs packet characterization based on channel information.Based on the channel number, the channel configuration characterizationstage determines: (i) the amount of 16-bit quantities to skip at thestart of the packet to get to the true start of the packet (skips anyMAC prepend or external switch prepend); and (ii) determines whether thepacket is required to have General Framing Procedure (GFP) framing,Point-to-Point Protocol (PPP) framing, or Ethernet framing. The outerencapsulation characterization stage 331 performs packetcharacterization based on the framing type determines by the stage 330.Based on the framing type determined in stage 330, the outerencapsulation characterization stage 331 generates: (i) an Ethernetencapsulation type and a start offset of the Ethernet packet; or (ii) aninner packet type and a starting offset into the packet of the innerpacket (e.g. IPv4 at starting offset 14). Ethernet encapsulationcharacterization stage 332 performs Ethernet characterization if theprevious stages indicate that the packet is an ethernet packet. Stage332 can skip various field types and VLAN tags included in the packet.Stage 322 generates information regarding the Ethernet encapsulation,the inner packet type, and a starting offset into the packet of theinner packet. The inner packet characterization stage 333 characterizesthe inner packet depending on the inner packet type determined by theouter encapsulation characterization stage 331 or the Ethernetcapsulation characterization stage 332. The inner packetcharacterization stage 333 generates information about the inner packetheader, and an indication of the starting offset of the payloadencapsulated within the inner packet. The characterization valuesresulting from characterizer 200 are illustrated in FIG. 9. In thisfashion, characterizer 200 is utilized to perform a first level ofpacket characterization in high speed combinational logic. A secondlevel of packet classification is performed by classifier 201. Thecharacterization value is communicated to a picoengine in the picoenginepool included in the classifier 201.

The classifier 201 is an instance of the picoengine pool as describedin: 1) U.S. patent application Ser. No. 14/267,298, entitled“Kick-Started Run-To-Completion Processor Having No InstructionCounter”, filed May 1, 2014, by Gavin J. Stark; and 2) U.S. patentapplication Ser. No. 14/311,222, entitled “Skip Instruction To Skip ANumber Of Instructions On A Predicate”, filed Jun. 20, 2014, by Gavin J.Stark (the subject matter of these two patent documents is incorporatedherein by reference). A detailed diagram of a picoengine pool isprovided in FIG. 14. As explained in those patent documents, thepicoengines of the picoengine pool are very small run-to-completionprocessors that and do not fetch instructions on their own (withoutexternal prompting or without being prompted to by having executed afetch instruction), and that do not have instruction counters.Accordingly, the classifier circuit 115 does not include any processorthat fetches instructions (without being prompted to by an externaltrigger or without being instructed to by having executed a specificfetch instruction), and that does not have an instruction counter, andthat analyzes packets. This classifier circuit, however, doescharacterize and classify the incoming packet, thereby generatingmetadata 207 about the packet. The resulting metadata 207 can includeinformation about the packet, information about the flow to which thepacket belongs, information about how the packet was initiallyprocessed, and so forth. As explained in further detail below, if theport is in a novel “command mode”, then the classifier 115 interpretsthe incoming packet as a special command packet, and orchestratescarrying out the operation called for by the opcode of the commandpacket, and if necessary generates a special command mode command packetthat is then output from an appropriate egress port of the NFX. FIG. 15illustrates a sequence of instructions that can be executed by apicoengine. The execution of a sequence of instructions by a picoenginewithin a picoengine pool as described in: 1) U.S. patent applicationSer. No. 14/267,342, entitled “TABLE FETCH PROCESSOR INSTRUCTION USINGTABLE NUMBER TO BASE ADDRESS TRANSLATION”, filed May 1, 2014, by GavinJ. Stark; (the subject matter of this patent document is incorporatedherein by reference). In one example, the CCTC configures a picoenginewithin the picoengine pool to perform a first sequence of instructionswhen the analysis type is set to a first value, and configures thepicoengine of the picoengine pool to perform a second sequence ofinstructions when the analysis type is set to a second value.

CCTC 115 writes to the various memories located in corner A 105 viaconnections 141 and connections 139 (address, data, control). Logic (notshown) included in CCTC 115 generates the address, data, and controlsignals based on output data from a picoengine. This logic is not shownbut standard registered IO architecture can be used.

In addition to generating metadata, the classifier outputs an outputdata value 208, a template value 219, and start of packet, end of packetand length of packet data 206. These outputs are provided to tablelookup and mux circuit (TLMC) 202. Table lookup and mux circuit 202includes a template RAM 221, mux A 222, mux B 223, a reduce table A 224,and a reduce table B 225, and a multiplexer circuit 226. The templateRAM 221 has multiple inputs including an address input, a data input,and a read/write input. In operation the output data reader 233 of theclassifier 201 can write a template value 219 into the template RAM 221and can supply an address to the template RAM 221 so to cause thetemplate RAM 221 to output multiple bits to multiple multiplexercircuits and both reduce tables 224 and 225. In this fashion, theclassifier is able to both program the template RAM 221 as well asselect one of a plurality of template values stored in the template RAM221. Selecting different template values will 1) change the inputs tothe multiple multiplexer circuits, thereby adjusting which inputterminal of a multiplexer circuit is coupled to the multiplexercircuit's output terminal, and 2) will change the lookup performed byeach reduce table. The contents of template RAM 221 are illustrated inFIG. 10.

The output data value 208 is coupled to a first input of the multiplexercircuit 226, a first input of the mux A 222, and a first input of themux B 223. The metadata 207 is coupled to a second input of themultiplexer circuit 202, a second input of the mux A 222, and a secondinput of the mux B 223. The start of packet—end of packet—length ofpacket data 206 is coupled to a fourth input of the multiplexer circuit226. The output of mux A 222 is coupled to an input of reduce table A224 and the output of the reduce table A 224 is coupled to a fourthinput of the multiplexer circuit 226. In a similar fashion, the outputof the mux B 223 is coupled to an input of Reduce table B 225 and theoutput of the Reduce table B 225 is coupled to a fifth input of themultiplexer circuit 226.

In operation, the classifier controls (via the template value in thetemplate RAM 221) (i) whether the output data value 208 or the metadata207 is input to Reduce table A 224 through mux A 222, (ii) whether theoutput data value 208 or the metadata 207 is input to Reduce table B 225through mux B 223, (iii) the type of lookup algorithm is performed byreduce table A 224, (iv) the type of lookup algorithm is performed byreduce table A 225, and (v) how the 6-bit compressed bits will be placedwithin a flow ID. The term “flow ID” (also “flow Id” or “Flow Id”)stands for “flow identifier value” or “flow identification value”. Theflow ID is a 256-bit value that is unique to the packet flow to whichthe packet belongs. The flow ID includes certain packet headers as iswithout any modification and “compressed” (reduced number of bits)versions of other packet headers. Other packet headers are omitted alltogether and are not included into the flow ID at all (e.g. due toknowledge of how the NFX integrated circuit is connected the omittedpacket headers do not contain any useful information).

The Flow ID does not include any wildcards. The Flow ID is not selectbits from only one field of the packet header. In one example, selectedbits are taken from the IP source field of the packet header, the IPdestination field of the packet header, the TCP source port field of thepacket header, the TCP destination port field of the packet header, andapplication layer protocol. Mux A 223 is shown in greater detail in FIG.11. Mux A 222 includes multiple groups of multiplexers 260-263. Eachgroup of multiplexers includes eight 48:1 multiplexers. The first groupof multiplexers 260 receives output data [0:255] and metadata [0:127],the switching of which is controlled by ASEL[42:47] output by thetemplate SRAM 221. Each bit of ASEL[42:47] is connected to one of theeight multiplexers. The second group of multiplexers 261 receives outputdata [0:255] and metadata [0:127], the switching of which is controlledby ASEL[36:41] output by the template SRAM 221. The third group ofmultiplexers 262 receives output data [0:255] and metadata [0:127], theswitching of which is controlled by ASEL[30:35] output by the templateSRAM 221. The last group of multiplexers 263 receives output data[0:255] and metadata [0:127], the switching of which is controlled byASEL[0:5] output by the template SRAM 221.

Reduce table A 224 generates a reduced MAC destination address. Thereduce table A 224 is showing greater detail in FIG. 12. Reduce table A224 includes a barrel shifter 272, a programmable mask circuit 273,adder circuit 289, SRAM 274, ECC correction circuit 275, a programmablelookup circuit 276, and a configuration table 277. In operation, reducetable code 279 is received by the configuration table 277 from thetemplate RAM and causes the configuration table 277 to outputconfiguration bits 280. Different portions of the configuration bits 280are coupled to different circuits within the Reduce table A 224. In oneexample, configuration bits 280 include 32 bits, six of which areidentified as a number of bits to shift 278, four of which areidentified as a number of bits to mask 281, fourteen of which areidentified as a base address 282, and eight of which are identified asan algorithm select value 283.

Either the output data value 208 or the metadata 207 is input viaconnections 272 to barrel shifter 272 from mux A 222 as shown in FIG.11. FIG. 24 illustrates the input data input to reduce table A. Theinput data includes 16 don't care bits and 48 MAC source address bits.The top 14 bits of the MAC source address need to be bottom-aligned soas to be aligned for the programmable mask circuit 273. The bottom 34bits of the MAC source address need to be top-aligned so as to bealigned to for lookup algorithm 276. Barrel shifter 272 rotates theinput data according to the number of bits to shift value 278 to providethe desired alignment. FIG. 25 illustrates the bit alignment of the dataoutput from barrel shifter 272. As desired, the 34 bits of MAC sourceaddress are top-aligned and the fourteen bits of MAC source address arebottom-aligned. The shifted data is coupled from barrel shifter 272 toprogrammable mask circuit 273 and programmable lookup circuit 276. Inthe specific example illustrated in FIG. 12 only the top 34 bits of theshifted data 292 are provided to the programmable lookup circuit 276.The programmable mask circuit 273 includes a decoder 290 that generatesfourteen binary output signals (“barrel shifter output data”) inresponse to receiving the number of bits to mask value 281. Each of thefourteen binary output signals is coupled to a first input of an ANDgate. A second input of each AND gate is coupled to receive one of thebottom 14 bits of the shifted data. The programmable mask circuit 273thereby outputs a fourteen bit masked value (“masked barrel sifteroutput data”) to the input of adder 289. The adder 289 performs abit-wise addition of the 14 bit masked value and the base address value282 and outputs the results (“a set of address bits”) to SRAM 274. Thecombination of barrel shifter 272, programmable mask circuit 273 andadder 289 is referred to herein as “a programmable modifier” 310.

SRAM 274 includes both first lookup algorithm contents 286 and secondlookup algorithm contents 287. The result from adder 289 serves as anaddress into SRAM 274. In response to receiving the result from adder289 the SRAM 274 outputs a reference value to ECC correction 275. Thereference value is then output to programmable lookup circuit 276. Thetype of algorithm performed by programmable lookup circuit 276 isdetermined by the algorithm select value 283. In one example theprogrammable lookup circuit 276 can perform a CAMR 34 lookup algorithm.The programmable lookup circuit 276 is an instance of the CAMR 32 lookupalgorithm as described in: 1) U.S. patent application Ser. No.13/598,448, entitled “TRANSACTIONAL MEMORY THAT PERFORMS A CAMR 32-BITLOOKUP OPERATION”, filed Aug. 29, 2012, by Gavin J. Stark; (the subjectmatter of which is incorporated herein by reference). A 6-bit outputvalue is output by the lookup algorithm 276 after performing the CAMR 34lookup algorithm. The 6-bit output value is a compressed or “reduced”version of the 64-bit input the data value 271. FIG. 26 illustrates theoutput value from reduce table A. The 6 bits of reduced MAC destinationaddress are bottom-aligned and padded with 26 “0”s. The output value iscoupled to the multiplexer circuit 226.

Reduce table B 225 generates a reduced VLAN ID. The structure of reducetable B 255 is the same as the structure illustrated in FIG. 12 forreduce table A 224. FIG. 27 illustrates an example of input data toreduce table B 225. The input data is either output data value 208 orthe metadata 207. In the present example, the input data includes 36don't care bits that are top-aligned, 12 bits of VLAN ID, and 16 moredon't care bits that are bottom-aligned. In operations, tableconfiguration circuit sets configuration bits such that the algorithmselect value is set to “pass through” mode. In pass through mode, theprogrammable lookup circuit automatically outputs one of the pluralityof 6-bit values read from SRAM. A barrel shifter included in reducetable B 225 rotates the input data such that the VLAN ID isbottom-aligned as shown in FIG. 28. The 12 bits of VLAN ID are masked bythe programmable mask circuit and are added to a base address value,thereby generating a set of address bits. The set of address bits isused to read a data value from an SRAM memory. The data value read fromthe SRAM memory includes one or more 6-bit values. In one example, basedon the programmable lookup up circuit configuration, the programmablelookup circuit simply outputs the first of the data values read from theSRAM memory. FIG. 29 illustrates the output of reduce table B 225. Theoutput from reduce table B includes 8 bits of reduced VLAN that arebottom-aligned and 24 “0” bits padding the remainder of the 32-bitoutput value.

FIG. 13 illustrates the multiplexer circuit 226 in greater detail.Multiplexer circuit 226 includes 32 groups of multiplexers 295-299. Eachgroup of multiplexers includes eight 58:1 multiplexers. Each group ofmultiplexers receives output data [0:255] via connections 294, reducetable A output value [0:31] via connections 288, reduce table B outputvalue [0:31] via connections 300, start of packet, end of packet, andlength of packet [0:8] via connections 301, and metadata [0:127] viaconnections 302. The first group of multiplexers 295 also receivesMUXSEL [192:197] via connections 303. The second group of multiplexers296 also receives MUXSEL [186:191]. The third group of multiplexers 297also receives MUXSEL [181:186]. The fourth group of multiplexers 298also receives MUXSEL [175:180]. The last group of multiplexers 299 alsoreceives MUXSEL [0:5]. The first group of multiplexers 295 output theanalysis type value. The remaining 31 groups of multiplexers 296-299combine to output 32 bytes of the flow ID via connections 307. In thisfashion the value stored in the template RAM 221 controls the flow IDoutput by the multiplexer circuit 226. FIG. 30 illustrates the contentsof each byte of the flow ID. The analysis type is stored in byte 0, thecompressed ethernet source address is stored in byte 1, the compressedVLAN ID is stored in byte 2, the IP transport protocol is stored in byte3, the IP source address is stored in bytes 4-7, the IP destinationaddress is stored in bytes 8-11, the source port is stored in bytes12-13, the destination port is stored in bytes 14-15, and “0s” arestored in remaining bytes 16-31.

Referring back to FIG. 5, the CCT circuit 115 outputs the flow ID 117 toCCT circuit 118. In the present example CCT circuit 118 simply passesthe flow ID through to the input of the exact-match flow table structure121 as Flow ID 119. Flow IDs 117 and 119 are also referred to here as a“flow key”.

Exact-match flow table structure 121 is a circuit that includes a hashgenerator circuit 122 and a memory and associated hash lookup circuitry.The exact-match functionality of exact-match flow table structure 121 isillustrated in FIG. 6. The exact-match flow table structure does not andcannot store a flow ID that has wildcards. The exact-match flow tablestructure does not include any information from a particular field of anincoming packet to which a value is to be matched. There is only 1-bitof SRAM that is used to store any single bit of a flow ID.

The memory maintains and stores the flow table 90 of the corner portion,where the flow table 90 includes a set of hash buckets, and where eachhash bucket includes a set of entry fields. The illustrated hash bucket123 is shown in the diagram as a row of boxes. The second entry field ofthe bucket is identified by reference numeral 124. The hash generator122 generates a hash value 125 from the flow ID 119, and supplies thehash value 125 to the flow table circuitry. The hash value points to oneof the hash buckets. In response, the flow ID stored in each entryfields of the selected bucket is supplied to a different comparatorcircuit 315-318. Each comparator circuit also receives the input flow ID119. Each comparator determines if the flow ID received from each of theentry fields in the flow table is an exact-match of input flow ID 119.The output from each comparator circuit 315-318 is grouped together toform a multi-bit select line that is coupled to input leads of encoder319. Encoder 319 outputs a single bit match value that indicates if aflow ID exact-match was found by comparators 315-319. Encoder 319 alsooutputs a multi-bit select value to multiplexer 320. In the presentexample, multiplexer 320 includes 403 5-to-1 multiplexers. If anexact-match is found, then the entire flow entry from which the matchingflow ID was stored is output on the output terminals of multiplexer 320.If an exact-match is not found, a default flow entry is output bymultiplexer 320.

FIG. 31 is a diagram that illustrates the values (a “flow entry”) thatare stored in one entry field of a hash bucket of the flow table if thatentry field is occupied (occupied by a “flow entry”). The first 256-bitvalue in any occupied entry field of the hash bucket are tested todetermine if one of the entry fields stores the 256-bit flow ID value119. If it does, then there is said to be a “hit”. If there is no match,then there is said to be a “miss”. For there to be a hit, the 256 bitsof the incoming flow ID must match exactly the 256 bits of the first256-bit “flow ID” field at the beginning of the overall flow entry.

Assuming in this example that there is a “hit”, then the remainingvalues (see FIG. 31) of the flow entry stored in the entry field areoutput as a match value (the number of the flow entry), an ingressaction value, an egress action value (including an 8-bit code thatidentifies an egress script), a 24-bit value that represents an egressflow number, an egress port value (including 6-bits that indicate anegress port and 10-bits that indicate particular NFX), a statisticsvalue (including twenty-nine bits that indicate the number of packetsthat matched this entry and thirty-five bits that indicate the number ofbytes that match this entry), and a priority value. The action may, forexample, be an instruction to output the corresponding packet onto aparticular egress port of a particular corner portion of the NFXcircuit.

The actual packet, rather than being passed to the flow table structure121, is buffered in buffer SRAM block 126. An instruction andinformation appropriate to cause the looked-up action to be carried outis loaded into the queue for the egress port (the egress port from whichthe packet is to be output from the NFX circuit in accordance with thelooked-up action in this example). There is one such queue in block 110for each egress port of the NFX circuit. At an appropriate time asindicated by the scheduler 111, the crossbar circuit 112 is controlledby the scheduler 111 to direct the packet (which is now being output bythe buffer SRAM 126 to the crossbar circuit 112) through the crossbarswitch circuitry to the particular FIFO 128 for the indicated egressport. The crossbar communication is illustrated in FIG. 34. The first 8bits of the crossbar communication is an 8-bit code that identifies anegress script. The next twenty-four bits of the crossbar communicationis a 24-bit egress flow number. The next sixteen bits of the crossbarcommunication is a 16-bit egress port number which is only used whensending the crossbar communication through another NFX. The remainder ofthe crossbar communication is the original packet excluding its originalethernet header.

The egress FIFO block 127 includes one such FIFO for each of the egressports of the NFX circuit. The packets buffered in the FIFO 128 aresupplied to egress modifier circuit 147. Egress modifier circuit 147includes three memories 148-150, an input buffer 151, and an egresspacket modifier 152. FIG. 16 illustrates the egress modifier circuit 147in greater detail. For a detailed description of the inner workings ofthe egress packet modifier circuit 147 see: U.S. patent application Ser.No. 14/941,494, entitled “Script-Controlled Egress Packet Modifier”,filed Jul. 14, 2013, by Chirag P. Patel and Gavin J. Stark (the subjectmatter of which is incorporated herein by reference).

Memory 148 is an egress packet script memory that is 64 bits wide andstores instruction sets (also referred to as “scripts”). Memory 149 is afirst egress packet modifier memory that is 256 bits wide and storesnon-flow specific header information. Memory 150 is a second egresspacket modifier memory that is 256 bits wide and stores flow specificheader information. Memories 148-150 are written to via connections 139from CCT circuit 115. A script parser 409 included in the egress packetmodifier 152 receives an 8-bit script code 410 and a 24-bit egress flownumber 426 that was communicated across the crossbar along with thepacket. By adding the appropriate script code to the front of thepacket, the corner of the NFX determines the type of egress processingthat will later be performed on the packet by the egress packetmodifier. In operation, the script parser 409 generates a first memoryaddress based on the 8-bit script code received from packet supplymemory (FIFO) 411 and sends a read request to the egress packet scriptmemory 148 that includes the first memory address. The script parser 409also generates a second memory address based on the 8-bit script codeand sends a read request to the first egress packet modifier memory 149that includes the second memory address. The script parser 409 furthergenerates a third memory address based on the 24-bit egress flow number426 received from packet supply memory (FIFO) 411 and sends a readrequest to the second egress packet modifier memory 150 that includesthe third memory address. In response to the three read requests, thescript parser 409 receives one set of instructions (“script”) 404, oneset of non-flow specific header information 406, and one set of flowspecific header information 408. An example of a set of non-flowspecific header information read out of the first egress packet modifieris illustrated in FIG. 35. The set of header information read out of thefirst egress packet modifier memory includes an IP version checksumvalue, a transport protocol value, and VXLAN network ID value, andethernet destination address, and an ethernet source address. The scriptparser 409 reads a second set of header information 408 from the secondegress packet modifier memory 150 in response to sending a read request407. An example of a set of flow specific header information read out ofthe second egress packet modifier memory is illustrated in FIG. 36. Theset of arguments read out of the second egress packet modifier memoryincludes an ethernet destination address, an ethernet source address,and IP source address, and ethernet transport protocol, and IP transportprotocol, a transport protocol, and an ethernet transport protocol.

The script parser 409 also receives a clock input signal that is sharedwith pipeline 7. Script parser 409 outputs multiple opcodes (OPCODES1-8). Pipeline 7 includes multiple stages 414-417. All eight opcodesgenerated by the script parser 409 are supplied to stage 1 along with 32bytes of packet data from packet supplying memory 411. Each stage of thepipeline 7 adds a portion of header information read from the egresspacket modifier memories to the packet data. The combination of thescript parser 409 and pipeline 7 is referred to herein as a “scriptinterpreter”.

The 256-byte minipacket 420 is supplied to an egress MAC circuit 129.For additional information on egress MAC circuit 129, see: U.S. patentapplication Ser. No. 14/321,732, entitled “Merging PCP Flows As They AreAssigned To A Single Virtual Channel”, filed Jul. 1, 2014, by Joseph M.Lamb (the subject matter of which is incorporated herein by reference).The packet is then output from the egress port of the NFX as indicatedby the flow entry in the flow table 90. As described in further detailbelow, the NFP circuit 53 controls how the NFX circuits 49-52 switch theincoming packets onto output ports of the SDN switch 21. The NFP circuit53 does this by loading appropriate values into the flow tables of theNFX circuits.

A set of values in an entry field of a flow table of an NFX circuit isalso referred to as a “flow entry”. An incoming packet received onto theSDN switch 21 via one of the NFX circuits may be switched so that it isoutput from the SDN switch 21 via an NFX circuit. Alternatively, or inaddition, an incoming packet received onto the SDN switch 21 via one ofthe NFX circuits may be switched so it is supplied across one of thenetwork links to the NFP circuit 53. How the packet is switched dependson the type of packet received, and how the flow tables of the NFXcircuits are loaded.

If there is a “miss”, then there was no flow entry in the flow tablestructure 121 whose “flow ID” field matched (matched exactly) the “flowID” of the incoming packet. In this case, the flow table structure 121takes a default action. The default action is to forward the packet tothe NFP circuit using a special command packet format. The NFPinitializes the flow table structure of each NFX circuit so that thedefault action will generate a proper command packet with the propercommand header or headers, and so that this proper command packet willbe output from the NFX onto the correct NFX egress port such that thecommand packet will be make its way to the NFP.

FIG. 17 illustrates data center 10 of FIG. 1 in greater detail. Aspecific example of a Web server operating on a virtual machine (VM A1)on a host server device 22 located around 11 switched by a SDN switch 21which communicates with a gateway 23 via a spine 16 is described herein.FIG. 18 illustrates a first instance in which an HTTP request iscommunicated from the Internet 18 to the virtual machine VM A1 runningon host server device 22 (communication #1). This communication is afive tuple communication packet not including any tags. This type ofcommunication is referred to herein as a “packet type A” packet. Thegateway 20 includes a network address translator that converts theincoming packet carrying the HTTP get request from a packet type A to apacket to B by adding a VXLAN header to the packet. The packet is thencommunicated from the Gateway 20 through spine 16 to SDN switch 21 (alsoreferred to herein as a “leaf” and a “top of rack switch”). The SDNswitch 21 converts the packet from a packet type B to a packet type C,wherein a packet type C does not have a VXLAN header, but rather a VLANheader. The conversion from packet type B to a packet type C isillustrated in FIG. 19. As is shown in FIG. 19 the entire VXLAN headeris removed by the SDN switch 21. The ethernet destination address ischanged from the leaf (SDN switch 21) ethernet address to the serverethernet address. The ethernet source address is changed from thegateway ethernet address to the leaf ethernet address. Four fields areadded: (i) a VLAN virtual LAN number, (ii) an ethernet transportprotocol (e.g. IPV4), (iii) a checksum value, and (v) an IP transportprotocol (e.g. TCP). The IP destination address is changed from thepublic IP address from which the HTTP request originated to the IPaddress of the leaf for virtual machine. All other packet fields remainunchanged. After this transformation from a packet type B packet to apacket type C packet, the HTTP request is communicated to virtualmachine VM A1 running on host server device 22.

FIG. 20 illustrates a scenario in which the website application runningon the host server device 22 needs to access information stored onvirtual machine VM A2 running on host server device 23 before serving upthe request webpage (e.g. accessing login information or an image file)(communication #2). To access information stored on virtual machine VMA2s, the Web server running on VM A1 has to send a packet type C (VLANpacket) to SDN switch 21 to access a virtual machine running outside ofhost server device 22. SDN switch 21 will in turn communicate a packettype C to the virtual machine VM A2 running on host server device 23within rack 11. Given that the packet type communicated from virtualmachine VM A1 to virtual machine VM A2 is the same packet type C, theSDN switch 21 does not need to modify the packet type. The similarscenario in which virtual machine VM A2 responds with the requested datato virtual machine VM A1 (communication #3) also does not require anypacket modification.

FIG. 21 illustrates a fourth scenario in which the Web server running onthe virtual machine VM A1 running on host server device 22 generates aresponse and communicates the response from the virtual machine VM A1 tothe SDN switch 21 as a packet type C communication. SDN switch 21 has toconvert the packet type C communication to a packet type B communicationin order to communicate data to gateway 20 through spine 16. Theconversion from packet type C to a packet type B is illustrated in FIG.22 (including FIG. 22A and FIG. 22B). The VLAN number is removed fromthe packet. The ethernet destination address is changed from the leaf(SDN switch) ethernet address to the gateway ethernet address. Theethernet source address is changed from the server ethernet address tothe leaf (SDN switch) ethernet address. A VXLAN header is also added tothe packet. The VXLAN header includes: an ethernet destination addressthat is the ethernet addresses the spine 16, and the data source addressthat is the ethernet address of the leaf (SDN switch 16), and IP sourceaddress that is the IP address of the leaf (SDN switch 16), and ethernettransport protocol (e.g. IPV4), and IP version length checksum, and IPtransport protocol (e.g. TCP), and IP destination address that is the IPaddress of the gateway 20, a transport protocol (e.g. UDP), and a VXLANnetwork ID (e.g. overlay network A).

FIG. 23 (including FIG. 23A and FIG. 23B) is a flowchart 500illustrating VXLAN encapsulation flow routing in a leaf (SDN switch). Instep 501, it is determined that the transfer protocol of the receivedpacket is IPV4. If the packet is IPV4 the flow moves a step 502. In step502 it is determined if no IP header options are present in the receivedpacket. If no IP header options are present in the received packet theflow moves to step 503. In step 503 it is determined that the IP headerchecksum is proper. If the IP header checksum is proper the flow moves astep 504. In step 504, it is determined that the header indicates a TCPor UDP transport protocol. If any of steps 501 through 504 are not metthe flow moves to step 505. In step 505 the template RAM outputstemplate number zero and the flow moves to step 506. In step 506, theflow ID 0 is used to perform a lookup operation and the flow moves tostep 507. In step 507, the packet is forwarded to the NFP (where the NFPwill determine how to route the packet). Returning to step 504 if theheader does indicate a TCP or UDP transfer protocol the flow moves tostep 508. In step 508 the template RAM outputs template number one. Instep 509, the ethernet source address is compressed using a first reducetable and the VLAN is compressed using a second reduce table. In step510, a flow ID including the compressed ethernet source address and thecompressed VLAN is generated. This can be accomplished by mapping usinga template. In step 511, the flow ID is used for forming lookupoperation. In step 512, it is determined if the lookup operationresulted in a miss or hit. If the lookup operation resulted in a miss,then the flow moves to step 507 and the packet is forwarded NFP wherethe NFP will determine how to route the packet. If the lookup operationresulted in a hit, then the flow moves to step 513. In step 513, thefollowing four items are forwarded across the crossbar of the egressmodifier: 1) the lookup egress action code; 2) the flow number; 3) theegress port number; and 4) the payload of the original incoming frame.In step 514, it is determined if the egress action code is three. If theegress action code is three the flow moves onto step 515. In step 515,the egress modifier performs the egress action indicated by the egressaction code thereby adding the MAC header to the payload and thenencapsulating with a VXLAN header to form a packet. In step 517, thepacket is transmitted out of the egress port indicated by the egressport number. Going back to step 514, if the egress action code is notthree, the flow moves to step 516. In step 516, the egress modifierperforms the egress action indicated by the egress action code therebyadding a VLAN field and a MAC header onto the payloads so as to form apacket. Then the flow moves to step 517, wherein the packet istransmitted out of the egress port indicated by the egress port number.It is noted that in one example steps 501 through 504 are performed bythe picoengine included in CCT circuit 115.

FIG. 32 illustrates a first bit-by-bit exact-match using an exact-matchflow table. Integrated circuit 500 includes a plurality of QSFP 40 Gbpsmodules 551-560, a bit-by-bit exact-match logic block 568, and an SRAM563. SRAM 563 includes an exact-match flow table 564 that storesmultiple flow entries such as flow entry 565. Each flow entry includes aflow ID 562, an egress action 566, and egress port 567. In operation, anincoming packet 561 is received by the integrated circuit 500 via module553. In response to receiving incoming packet 561, integrated circuit500 generates a flow ID 570. Integrated circuit 500 compares thegenerated flow ID 570 with flow ID 562 included in flow entry 565 usingthe bit-by-bit exact-match logic 568. In a first scenario it isdetermined that the flow ID 570 matches flow ID 562 in which caseintegrated circuit 500 performs the egress action and communicates theincoming packet out of the appropriate egress port. In a second scenarioit is determined that the flow ID 570 does not match flow ID 562 inwhich case integrated circuit 500 sends the incoming packet 561 to anexternal device that generates a new flow entry 569 that is communicatedback to integrated circuit 500 and is loaded into the exact-match flowtable 564 stored in SRAM 563.

FIG. 33 illustrates a second bit-by-bit exact-match using an exact-matchflow table. Integrated circuit 600 includes a first part of theintegrated circuit 601 and a second part of integrated circuit 602. Inone example, each part is referred to as a separate island in amulti-island architecture. The first part 601 of integrated circuit 600includes plurality of QSFP 40 Gbps modules 603-612, a bit-by-bitexact-match logic block 620, and an SRAM 615. SRAM 615 includes anexact-match flow table 616 that stores multiple flow entries such asflow entry 617. Each flow entry includes a flow ID 618, an egress action619, and an egress port 620. In operation, an incoming packet 613 isreceived by the first part 601 of integrated circuit 600 via module 605.In response to receiving incoming packet 613, the first part 601 ofintegrated circuit 600 generates a flow ID 614. The first part 601 ofintegrated circuit 600 compares the generated flow ID 614 with flow ID618 included in flow entry 617 using the bit-by-bit exact-match logic620. In a first scenario it is determined that the flow ID 614 matchesflow ID 618 in which case the first part 601 of integrated circuit 600performs the egress action and communicates the incoming packet out ofthe appropriate egress port. In a second scenario it is determined thatthe flow ID 614 does not match flow ID 617 in which case the first part601 of integrated circuit 600 sends the incoming packet 613 to thesecond part 602 of integrated circuit 600 that generates a new flowentry 621 that is communicated back to the first part 601 of integratedcircuit 600 and is loaded into the exact-match flow table 616 stored inSRAM 615.

In accordance with one novel aspect, an ingress network port of an NFXcircuit can operate in one of two modes: a data mode, or a novel commandmode. In the data mode, an incoming packet is handled as set forth abovein that the packet is characterized and classified and then switched inaccordance a flow entry in the flow table of the corner portion. Ifthere is a miss, then the packet is automatically forwarded to the NFPcircuit so that the NFP circuit can determine how the packet will beswitched out of the SDN switch. A packet includes a MAC preamble andstart frame delimiter, a MAC header portion, a MAC payload portion, anda MAC CRC portion. The ingress MAC circuitry 114 of FIG. 5 detects theMAC preamble and uses the MAC CRC to check that the intervening bitswere properly received, but the ingress MAC circuitry 114 does nototherwise use the intervening bits. All the network ports of the NFXcircuits that receive external network traffic onto the SDN switchand/or that output network traffic out of the SDN switch (i.e., do notlink to another NFX circuit) operate in this data mode.

In the novel command mode, the ingress network port receives a packet ofthe same form, but as illustrated in FIG. 37 the packet may contain aspecial command 653. The packet (MAC frame) passes through the sameingress MAC circuitry 114, but the subsequent CCT circuit 115 interpretsthe MAC payload portion of the frame to be a special command. Thecommand 653 includes header portion 650 (containing one or more commandheaders) and a payload portion 651. FIG. 38 is a diagram thatillustrates the form of one command header. The command header has aleading bit 660, followed by an 8-bit data portion 661, followed by a7-bit opcode portion 662. If the leading bit 660 of the first commandheader (first command header of the header(s) portion 650) is a “0”,then the command is a command to be carried out by receiving NFXcircuit. There is one and only one command header. If, however, theleading bit 660 of the first command header (first command header of theheader(s) portion 650) is a “1”, then the receiving NFX circuit is tostrip the leading command header from the packet, and to forward theremainder of the packet (in the form of the MAC frame, without theremoved command header) on to another NFX circuit as indicated by thedata portion of the command header. In this way, a command can beforwarded from one NFX circuit to the next, with each forwarding NFXcircuit popping off and removing the leading command header. The lastcommand header will have a leading bit of “0”, so that the NFX circuitthat receives this command detects this “0” and carries out theoperation specified by the opcode portion of the command. The operationmay, for example, be to add a flow entry into an NFX flow table. Theoperation may, for example, be to delete a flow entry from an NFX flowtable. The operation may, for example, be to output a packet carried bythe payload onto a particular egress port of the NFX circuit. Once theNFX circuits of the SDN switch are powered up and are properly set up bythe NFP circuit, all the network ports of the NFX circuits that coupledirectly to other NFX circuits are made to operate in this command mode.Intercommunication between NFX circuits uses this command mode.

FIG. 39 is a table that sets forth the operations specified by thevarious opcodes that may be present in a command. An opcode of “0000000”indicates that the receiving NFX circuit should strip the leadingcommand header, and forward the resulting packet out of the cornerportion and egress port identified by the data portion of the commandheader. The payload of the original incoming command is thereforeforwarded out of the NFX as the payload of the outgoing command, thedifference between the two commands being that the outgoing command hasone fewer command header (the leading command header of the incomingcommand is removed to make the outgoing command). An entire packet (forexample, a packet received onto the SDN switch that resulted in a “miss”in the receiving NFX circuit) can be forwarded through an NFX circuitusing this command, by including the packet as the payload portion ofthe command.

An opcode of “0000100” is an instruction to add a flow entry, where thepayload of the packet is the flow entry to be added.

An opcode of “0000101” is an instruction to delete a flow entry, wherethe payload of the packet identifies the particular flow entry to bedeleted by its flow ID. The classifier carries out the operations of theadd flow entry opcode and the delete flow entry opcodes by usingconnections 141 and 142 (see FIG. 5) to communicate with and control theflow table structure 121 so that the flow table 90 is changed asindicated by the command.

An opcode of “0000110” is an instruction to report status or statisticsfor a flow, where the particular flow ID identifying the flow is setforth in the payload of the command. Such status may, for example, bethe total number of packets of this flow that have been received, andthe total number of bytes that have been received of this flow. Thisstatistics information is maintained as part of the flow entry for theflow ID of the flow. The statistics are reported by the NFX sending acommand packet back to the NFP circuit. The command packet that is sentback carries the statistics information. To form this command packet,the called for statistics information from the flow table 90 is suppliedby connection 143 from the flow table structure 121 to the buffer SRAMblock 126 where the command packet to be output is formed, and in thisway the statistics information is made part of the outgoing commandpacket. The classifier that carries out the statistics operation ascalled for by the opcode forms the remainder of the packet complete withits command headers, and loads that part of the command packet viaconnections 144 into the buffer SRAM block 126.

An opcode of “0000111” is a “modeset” instruction to set a particularport on the NFX circuit to begin operating in the command mode, wherethe particular NFX port is given by the content of the data portion ofthe command.

An opcode of “0001000” is an instruction to: (i) configure a particularMAC circuit 114 or 129; (ii) load reduce SRAM of reduce tables 224 and225; and (iii) load one of the memories included in the egress modifiercircuit 147. The configuration data to be loaded into the MAC circuit iscarried in the payload of the command, where the particular MAC circuitto be configured is set forth in the data portion of the command. Inthis way, the speed of the MAC ports, the width of the MAC ports, andthe power levels of the MAC ports can be configured. Connections 139 and140 shown in FIG. 5 are used to communicate the configuration data fromthe classifier (that carries out the “000100” opcode) to the ingress MACcircuit to be configured.

Consider, for example, a situation in which the NFP circuit 53 needs toadd a flow entry into the flow table 105 of NFX circuit 52. Note thatNFP circuit 53 is not directly coupled by a single network connection orlink to NFX 52, but rather there is an intervening NFX circuit (eitherNFX 50 or NFX 49). Assume for purposes of this example that the ingressportion of the A1 network port of NFX circuit 50 is in the command mode,and further assume that the ingress portion of the A1 network port ofNFX circuit 52 is in the command mode. The NFP circuit 53 generates aMAC frame that contains a special command of the type set forth in FIG.35, where there are two command headers, and where the payload portionof the frame is the flow entry to be added. The leading command headerof the MAC frame is an instruction to the receiving NFX circuit toforward the packet out of its egress port C3 after stripping the leadingcommand header. The second command header has an opcode of “0000100” andis an instruction to the receiving NFX circuit, in this case NFX circuit52, to add the flow entry carried in the payload portion of the frameinto the flow table 105 as indicated by the data portion of the commandheader. This MAC frame, with the two command headers, is output fromnetwork port 88 of the NFP circuit 53 in ordinary fashion as an ordinaryMAC frame. The ingress portion of the A1 network port of NFX 50,however, receives the MAC frame and because the ingress portion is inthe command made, the ingress portion interprets the leading commandheader, and in accordance with the leading command header forwards theremainder of the MAC frame out of network port C3. Which port to outputthe MAC frame from is indicated by the data portion of the leadingcommand header. The resulting MAC frame at this point has only onecommand header, and this MAC frame with one command header is receivedonto the A1 port of NFX circuit 52. Because the ingress portion of theA1 network port is operating in the command mode, the command header isinterpreted. This command header has an opcode of “0000100”, so the flowentry contained in the payload of the MAC frame is added to the flowtable 105. Because the NFP circuit 53 stores information about theconfiguration and interconnectivity of the fabric of NFX switches, theNFP circuit 53 can generate a MAC frame with the appropriate commandheaders such that a desired command is supplied to the desired NFXcircuit, even if the NFP is not directly coupled to the NFX circuit towhich the ultimate command is directed. In this way, using suchcommands, the NFP circuit 53 controls the content of the flow tables ofthe NFX circuits and controls the operating mode of each NFX port. TheNFP can cause a command packet to be sent to any one of the NFXcircuits, where the command is an instruction to output the payload ofthe command in the form of a packet onto any one of the egress ports ofthe NFX circuit, and where the packet as output onto the egress port isactually a command packet itself.

Although certain specific embodiments are described above forinstructional purposes, the teachings of this patent document havegeneral applicability and are not limited to the specific embodimentsdescribed above. Accordingly, various modifications, adaptations, andcombinations of various features of the described embodiments can bepracticed without departing from the scope of the invention as set forthin the claims.

1-20. (canceled)
 21. A method comprising: (a) maintaining an exact-matchflow table on an integrated circuit, wherein the exact-match flow tablecomprises a plurality of flow entries, wherein each flow entry comprisesa Flow Identification value (Flow Id) and an action value; (b) receivinga first packet onto an integrated circuit; (c) analyzing the firstpacket and determining that the first packet is of a first type; (d) asa result of the determining of (c) initiating execution of a firstsequence of instructions by a processor of the integrated circuit,wherein execution of the first sequence causes bits of the first packetto be concatenated and modified in a first way thereby generating afirst Flow Id, wherein the Flow Id is of a first form; (e) determiningthat the first Flow Id generated in (d) is a bit-by-bit exact-match of aFlow Id of a first flow entry in the exact-match flow table; (f) usingan action value of the first flow entry in outputting packet informationof the first packet out of the integrated circuit; (g) receiving asecond packet onto an integrated circuit; (h) analyzing the secondpacket and determining that the second packet is of a second type; (i)as a result of the determining of (h) initiating execution of a secondsequence of instructions by the processor, wherein execution of thesecond sequence causes bits of the second packet to be concatenated andmodified in a second way thereby generating a second Flow Id, whereinthe second Flow Id is of a second form; (j) determining that the secondFlow Id generated in (i) is a bit-by-bit exact-match of a Flow Id of asecond flow entry in the exact-match flow table; and (k) using an actionvalue of the second flow entry in outputting packet information of thesecond packet out of the integrated circuit, wherein (a) through (k) areperformed by the integrated circuit.
 22. The method of claim 21, whereinthe executing of the first sequence causes a first set of the bits ofthe first packet to be concatenated and modified in the first waywhereas the executing of the second sequence causes a second set of thebits of the second packet to be concatenated and modified in the secondway.
 23. The method of claim 21, wherein both the first and secondpackets include a header of a particular type but the first Flow Idincludes at least one bit of the header of the particular type of thefirst packet whereas the second Flow Id includes no bit of the header ofthe particular type of the second packet.
 24. The method of claim 21,wherein both the first and second packets include a header of aparticular type but the first Flow Id includes a first modified form ofbits of the header of the particular type of the first packet whereasthe second Flow Id includes a second modified form of bits of the headerof the particular type of the second packet.
 25. The method of claim 21,wherein the integrated circuit comprises a plurality of substantiallyidentical portions and a crossbar switch, wherein each of thesubstantially identical portions is coupled to the crossbar switch,wherein each flow entry of (a) further includes an egress field, andwherein bits of the egress field identify one of the substantiallyidentical portions.
 26. The method of claim 21, wherein the integratedcircuit comprises a template memory and a multiplexer circuit, whereinthe template memory stores a plurality of template values, whereinexecution of the first sequence of (d) causes the template memory tooutput a first template value, wherein first bits of the first templatevalue are supplied onto select input leads of the multiplexer circuitsuch that the multiplexer circuit outputs the first Flow Id, whereinexecution of the second sequence of (i) causes the template memory tooutput a second template value, and wherein first bits of the secondtemplate value are supplied onto the select input leads of themultiplexer circuit such that the multiplexer circuit outputs the secondFlow Id.
 27. The method of claim 26, wherein second bits of the firsttemplate value determine what data is supplied onto data input leads ofthe multiplexer circuit in (d), and wherein second bits of the secondtemplate value determine what data is supplied onto the data input leadsof the multiplexer circuit in (i).
 28. The method of claim 21, whereinno flow entry of the exact-match flow table includes any wildcardindicator.
 29. The method of claim 21, wherein the exact-match flowtable is stored on the integrated circuit in a Static Random AccessMemory (SRAM).
 30. The method of claim 21, wherein the integratedcircuit comprises a programmable reduce table circuit, wherein theexecution of the first sequence of instructions causes the programmablereduce table circuit to perform a first type of lookup operation in thegeneration of the first Flow Id, and wherein the execution of the secondsequence of instructions causes the programmable reduce table circuit toperform a second type of lookup operation in the generation of thesecond Flow Id.
 31. An integrated circuit comprising: an amount ofmemory that stores a first sequence of instructions and a secondsequence of instructions; a processor system: 1) that executes the firstsequence of instructions thereby causing Flow Identification values(Flow Ids) to be generated from packets of a first type, and 2) thatexecutes the second sequence of instructions thereby causing Flow Ids tobe generated from packets of a second type, wherein the Flow Idsgenerated from packets of the first type are of a first form, andwherein the Flow Ids generated from packets of the second type of asecond form; and an exact-match flow table structure that maintains anexact-match flow table, wherein the exact-match flow table comprises aplurality of flow entries, wherein each flow entry comprises a Flow Idand an action value, wherein at least one of the Flow Ids of theexact-match flow table is of the first form, and wherein at least one ofthe Flow Ids of the exact-match flow table is of the second form. 32.The integrated circuit of claim 31, further comprising: acharacterization circuit that characterizes packets as being of one typeof a plurality of types.
 33. The integrated circuit of claim 31, whereinthe integrated circuit comprises a plurality of substantially identicalportions, wherein the amount of memory, the processor system, and theexact-match flow table structure are parts of one of the substantiallyidentical portions, wherein the integrated circuit further comprises acrossbar switch, and wherein the crossbar switch is coupled to each ofthe substantially identical portions.
 34. The integrated circuit ofclaim 31, wherein execution of the first sequence of instructions by theprocessor system causes Flow Ids to be generated by concatenating andmodifying packet bits in a first way whereas execution of the secondsequence of instructions by the processor system causes Flow Ids to begenerated by concatenating and modifying packet bits in a second way.35. The integrated circuit of claim 31, further comprising: aprogrammable reduce table, wherein the execution of the first sequenceof instructions by the processor system causes the programmable reducetable circuit to perform a first type of lookup operation, and whereinthe execution of the second sequence of instructions by the processorsystem causes the programmable reduce table circuit to perform a secondtype of lookup operation.
 36. The integrated circuit of claim 31,wherein the exact-match flow table is stored in a Static Random AccessMemory (SRAM).
 37. The integrated circuit of claim 31, wherein theexact-match flow table structure receives Flow Ids and for each receivedFlow Id determines whether the Flow Id is an exact-match for a Flow Idof a flow entry of the exact-match flow table.
 38. The integratedcircuit of claim 31, wherein the processor system comprises a singleprocessor that executes the first and second sequences of instructions.39. The integrated circuit of claim 31, wherein the processor systemcomprises a plurality of processors.
 40. A method comprising: (a)maintaining an exact-match flow table in an exact-match flow tablestructure on an integrated circuit, wherein the exact-match flow tablecomprises a plurality of flow entries, and wherein each flow entrycomprises a Flow Identification value (Flow Id), wherein the exact-matchflow table structure does not and cannot store a Flow Id having anywildcard indictor; (b) receiving a packet onto the integrated circuit;(c) analyzing the packet and determining that the packet is of one typeof a plurality of types; (d) as a result of the determining of (c)initiating execution of a selected one of a plurality of sequences ofinstructions, wherein execution of the selected one of the plurality ofsequence of instructions results in a generation of a FlowIdentification value (Flow Id); (e) supplying the Flow Id generated in(d) to the exact-match flow table structure; and (f) determining whetherthe Flow Id supplied in (e) is a bit-by-bit exact-match for the Flow Idof any flow entry stored in the exact-match flow table structure,wherein (a) through (f) are performed by the integrated circuit.